Method for generating an augmented set of images

ABSTRACT

A method for generating an augmented set of images involves data collection, data processing, and data augmentation processing performed to merge images. The data collection comprises the steps of choosing objects as selected objects, choosing configurations for imaging of the selected objects, and taking pictures with a second camera to create a set of raw object images. The data processing comprises the steps of performing dynamic range adjustment on the raw object image, performing color correction for corrected images, and removing uniform background from the corrected images to result in object images. The data augmentation processing performed to merge images comprises the steps of performing range simulation or magnification for resampled images, adding blur to the resampled images, adding noise to create final object images, and merging the final object images to the field images of a first camera to create a set of augmented images.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/966,630, filed on Jan. 28, 2020, the disclosure of which isincorporated herein by reference in its entirety.

GOVERNMENT INTEREST

The invention described herein may be manufactured, used, sold,imported, and/or licensed by or for the Government of the United Statesof America.

FIELD OF THE INVENTION

The present subject matter relates generally to generating a set ofaugmented images.

BACKGROUND OF THE INVENTION

Presently, artificial intelligence and machine learning (AI/ML) hasenabled the creation of software programs such as Neural Net (NN)algorithms that can be trained with training images to recognize objectsin new images. However, many images are needed to perform the training,and more unusual objects are not included in a large number of presentlyavailable image databases. Thus, a method of making large training setsof images is needed, which preferably does not involve simply taking newimages of the unusual objects, as this can be very costly and timeconsuming.

SUMMARY OF THE INVENTION

The present invention broadly comprises a method for generating anaugmented set of images. In one aspect, the method generates anaugmented set of images involving data collection, data processing, anddata augmentation processing performed to merge images. The datacollection comprises the steps of choosing objects as selected objects,choosing configurations for imaging of the selected objects, and takingpictures with a second camera to create a set of raw object images. Thedata processing comprises the steps of performing dynamic rangeadjustment on the raw object image, performing color correction forcorrected images, and removing uniform background from the correctedimages to result in object images. The data augmentation processingperformed to merge images comprises the steps of performing rangesimulation or magnification for resampled images, adding blur to theresampled images, adding noise to create final object images, andmerging the final object images to the field images of a first camera tocreate a set of augmented images.

In another aspect, the method includes measuring the optical andelectronic characteristics of a first camera, generating a set of rawobject images for a plurality of objects and a plurality ofconfigurations with a second camera, performing data processing on theraw object images to match image characteristics of the raw objectimages to image characteristics of the first camera to create finalobject images, and creating the augmented set of images by adding thefinal object images to field images collected with the first camera.

Yet, in another aspect, The method includes measuring the optical andelectronic characteristics of a first camera; generating a set of rawobject images for a plurality of objects and a plurality ofconfigurations with a second camera; and performing data processing onthe raw object images to match image characteristics of the raw objectimages to image characteristics of the first camera to create theaugmented set of images.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure of the present subject matter, includingthe best mode thereof, directed to one of ordinary skill in the art, isset forth in the specification, which makes reference to the appendedfigures, in which:

FIG. 1 illustrates a method in accordance with an embodiment of thepresent invention;

FIG. 2 shows a data collection method in accordance with an embodimentof the present invention;

FIG. 3 shows a data collection apparatus in accordance with anembodiment of the present invention;

FIG. 4 illustrates a data processing method in accordance with anembodiment of the present invention; and

FIG. 5 shows a data augmentation method in accordance with an embodimentof the present invention.

DETAILED DESCRIPTION

Reference is presently made in detail to exemplary embodiments of thepresent subject matter, one or more examples of which are illustrated inor represented by the drawings. Each example is provided by way ofexplanation of the present subject matter, not limitation of the presentsubject matter. In fact, it will be apparent to those skilled in the artthat various modifications and variations can be made in the presentsubject matter without departing from the scope or spirit of the presentsubject matter. For instance, features illustrated or described as partof one embodiment can be used with another embodiment to yield a stillfurther embodiment. Thus, it is intended that the present subject mattercovers such modifications and variations as come within the scope of thedisclosure and equivalents thereof.

To solve the above noted problem, the present inventors created a methodto create a large set of training images by incorporating images ofunusual objects into pre-existing background images. For example, in oneembodiment, a first camera, such as a GoPro® camera, may be used to takeimages in the field, and then a second camera, such as a Grasshopper®camera, can be characterized and used to take images of unusual objectsin a lab. The object images are then incorporated into the field imagesin various configurations to create augmented images. Therefore, a largeset of training images is created without taking a large set of imagesin the field.

In another embodiment, both the first and second cameras are sensorsthat detect radiation in the infrared (IR) spectrum, and the images areIR images.

FIG. 1 illustrates a method 10 for generating a set of augmented imagesaccording to a first embodiment of the present invention. Method 10includes the steps of data collection (step 100), data processing (step200), and data augmentation (step 300).

FIG. 2 shows the steps in the data collection step: choose objects (step110), choose configurations (step 120), and take pictures (step 130). Instep 110, objects that are desired to be added to the training datasetsare selected. Step 120 includes choosing all of the configurations forthe images of the selected objects, such as view angle of the camerawith respect to nadirs, aspect or rotational angles of the objectturntable 60 (shown in FIG. 3), lighting conditions, and aperture sizes.After all the configurations are selected, pictures are takencorresponding to all combinations of the selected configurations tocreate a set of raw object images.

The raw object images are taken of the objects against a uniformbackground dissimilar to the object, as shown in FIG. 3. Object 50 sitson turntable 60, which is located over uniform background 70. Turntable60 allows for images to be taken at a variety of rotational angles.Uniform background 70 allows for the background to be removed duringimage processing, as discussed later in step 230. In one embodimentusing visible light images, uniform background 70 may be a green screen.In another embodiment using IR images, uniform background 70 may be auniform temperature background.

FIG. 4 illustrates the data processing (step 200) that is performed onthe raw object images. Step 210 includes performing dynamic rangeadjustment on the raw object images. In one embodiment, two differentaperture sizes are used in order to vary the contrast in the images. Thehigh light level (larger aperture) images are adjusted with apredetermined gain value, calculated using the uniform background, whichis constant across all targets. In one embodiment, this gain is appliedto each image globally, but if more than 0.5% of the pixels aresaturated, the gain is decreased until the 0.5% threshold is met. Inorder to preserve intensity differences for a specific object, the samegain may be applied to the lower light level (smaller aperture) imagesof that object.

Step 220 then performs color correction. An X-Rite® color chart and aset of reflectance standards may be used as references for the colorcorrection, and these were captured by both cameras for eachenvironmental variation selected in step 120. Comparing the values ofthe XRite® color chart in both the first and second camera images, acolor correction matrix was created to transform the colors from thesecond camera and emulate the first camera.

In the embodiment where the first camera is a GoPro® camera, the colorcorrection is performed as follows. The GoPro® shifts color using asimple rational fraction equation to correct the color, as shown below.The p₁, p₂, and, q₁ variables were fit using the reflectance standards.

$y = \frac{{p_{1}x} + p_{2}}{x + q_{1}}$

Because color correction must be done on linear color images, the GoPro®image is linearized by solving the equation above for x and applying theresulting equation to the GoPro® images prior to color correction. TheGrasshopper® outputs linear data natively and requires no linearizationstep.

$x = {- \frac{p_{2} - {q_{1}*y}}{p_{1} - y}}$

For a linearized image reference, let C_(GP) be the 24×3 linear GoPro®X-Rite® color values with 3 columns representing the red, green and bluecolor channels, and C_(GH) similarly be the Grasshopper® XRite® colorvalues. The color correction matrix (M_(C)) is therefore:M _(C) =C _(GH) ⁻¹ C _(GP)

This color correction matrix provides color channel weights to shift theratio of the R, G, and B channels in a Grasshopper® image to the sameratios in the GoPro® image.

In another embodiment where the first camera is an IR spectrum sensor,step 220 matches an apparent temperature. In general, step 220 matches apixel intensity balance of the first camera.

Step 230 removes the uniform background from the images. In oneembodiment, to perform the background removal a mask may be created fromthe high light level (larger aperture) imagery by sectioning thebackground portions of the image using either HSV or L*a*b histogramanalysis and creating a binary mask file with background and objectregions. The background region may then be removed from the image usingthis mask, and a flat gray background may be inserted in its place. Themask used to remove the background region can be saved with the objectimage for any future processing or background insertion. It may alsoserve as labeling data for the location of the object in the image fortraining purposes.

FIG. 5 illustrates the data augmentation processing (step 300) that isperformed to merge the object images into the field images. Step 310performs range simulation or magnification. In one embodiment, range ormagnification variation includes at least 6 ranges, but sometimes up to10 depending on the object (smaller objects have additional highermagnification values). To simulate range variation, the second cameraimages were resampled using integer-based magnification. By forcinginteger values in the resampling, higher-order frequency effects such asedge ringing and other artifacts are avoided.

Range or magnification serves two different purposes. First, theindividual field of view in the second camera is made to match that ofthe first camera in order to have equal images. This allows applicationof the correct camera effects, which are defined in angular space, andthe sampling needs to be the same for that step.

Second, this generates images with varying magnification to simulatedifferent ranges that could not be collected in the lab. This is animportant part of the process, because it adds variety. This is done inaddition to and after matching the individual field of view of the firstcamera.

The sampling ratio between the input and output images is given by:

$s = \frac{f_{s_{in}}r_{out}}{f_{s_{out}}r_{in}}$

where f_(sin) and f_(sout) are the system sampling frequencies (incycles per radian), and r_(in) and r_(out) are the image magnificationsfor the second camera and the augmented image, respectively. In thiscase, r_(in)=1. To emulate the first camera, the input and output cameracharacteristics were taken from what is known about the second cameraand first camera, respectively. This includes characteristics like pixelpitch, pixel count, f/#, and FOV. The sampling ratio, f_(sin)/f_(sout),must be equal to or greater than one.

The above sampling ratio matches the sampling between the two cameras.In addition, more ranges are simulated by resampling at differentratios.

Once the resampling is completed, the images are padded to match thefirst camera format (for example, 3000×4000 pixels).

Step 320 adds blur to the images, and step 330 adds noise to the imagesto create final object images. The blur and noise variations aredesigned to account for potential environmental degradations like lowlight or dust. After the imagery was resampled and padded in step 310,the blur and noise are applied. A modulation transfer function (MTF) wasapplied to the image as a component of the presample MTF, which includesthe added blur, objective lens diffraction, and the detector MTF. Theapplication of the MTF was done in Fourier space.

White noise with user-defined RMS was then added to each color channelof the object images, individually. (This could be done for a camerahaving any number of channels, such as a multispectral camera.) In oneembodiment, the augmented dataset includes 2 noise levels and 3 blurlevels. The first noise and blur values (a blur value of 1 mrad andnegligible noise) are estimates of the first camera baseline values,while the other blur and noise values represent degradation of the firstcamera imagery.

Finally, step 340 adds the final object images to field images to createa set of augmented images. For example, 10,000 field images (taken withthe first camera) can be augmented with 4,000 object images (collectedwith the second camera) to create over 175,000 augmented images. Thismuch larger set of augmented images can then be used to train AI/MLalgorithms.

In another embodiment step 340 simply takes the final set of objectimages to be the augmented set of images. This allows images to be addedto a training database without being provided any field images. In thiscase, the images taken in step 130 would be modified as described toemulate a different camera (the first camera) than the camera used inthe method (the second camera).

The present written description uses examples to disclose the presentsubject matter, including the best mode, and also to enable any personskilled in the art to practice the present subject matter, includingmaking and using any devices or systems and performing any incorporatedand/or associated methods. While the present subject matter has beendescribed in detail with respect to specific embodiments thereof, itwill be appreciated that those skilled in the art, upon attaining anunderstanding of the foregoing may readily produce alterations to,variations of, and equivalents to such embodiments. Accordingly, thescope of the present disclosure is by way of example rather than by wayof limitation, and the subject disclosure does not preclude inclusion ofsuch modifications, variations and/or additions to the present subjectmatter as would be readily apparent to one of ordinary skill in the art.

It is obvious that many modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of the appended claims, theinvention may be practiced otherwise than as described.

What is claimed is:
 1. A method for generating an augmented set ofimages, the method comprising the steps of: data collection, comprisingthe steps of: choosing objects as selected objects, choosingconfigurations for imaging of the selected objects, and taking pictureswith a second camera to create a set of raw object images; dataprocessing performed on the raw object images, comprising the steps of:performing dynamic range adjustment on the raw object image, performingcolor correction for corrected images, and removing uniform backgroundfrom the corrected images to result in object images; and dataaugmentation processing performed to merge the object images of thesecond camera into field images of a first camera, comprising the stepsof: performing range simulation or magnification for resampled images,adding blur to the resampled images, adding noise to create final objectimages, and merging the final object images to the field images of afirst camera to create a set of augmented images.
 2. The method forgenerating an augmented set of images according to claim 1, wherein thefield images of the first camera are linearized for performing colorcorrection.
 3. The method for generating an augmented set of imagesaccording to claim 1, wherein a mask is used to remove the uniformbackground.
 4. The method for generating an augmented set of imagesaccording to claim 1, wherein the resampled images are resampled usinginteger-based magnification.
 5. The method for generating an augmentedset of images according to claim 1, wherein said configurations forimaging of the selected objects are chosen from a group comprising aview angle of a camera with respect to nadirs, aspect or rotationalangles of the object, lighting conditions, and aperture sizes.
 6. Amethod for generating an augmented set of infrared images, the methodcomprising the processes of: data collection, comprising the steps of:choosing objects as selected objects, choosing configurations forimaging of the selected objects, and taking infrared images with asecond infrared sensor to create a set of raw object images; dataprocessing performed on the raw object images, comprising the steps of:performing dynamic range adjustment on the raw object image, performingcorrection of apparent temperature for pixel intensity balanced images,and removing uniform background from the pixel intensity balanced imagesto result in object images; and data augmentation processing performedto merge the object images of the second infrared sensor into fieldimages of a first infrared sensor, comprising the steps of: performingrange simulation or magnification for resampled images, adding blur tothe resampled images, adding noise to create final object images, andmerging the final object images to the field images of a first infraredsensor to create an augmented set of infrared images.
 7. The method forgenerating an augmented set of infrared images according to claim 6,wherein taking infrared images with a second infrared sensor comprisesthe steps of: placing an object on a turntable, locating the turntableagainst a uniform temperature background, and generating a raw objectimages at a variety of rotational angles of the turntable against theuniform temperature background.
 8. A method for generating an augmentedset of images, the method comprising the steps of: measuring optical andelectronic characteristics of a first camera as image characteristics ofthe first camera; generating a set of raw object images for a pluralityof objects and a plurality of configurations with a second camera,wherein a raw object image is generated based on the steps of: placingan object on a turntable, locating the turntable against a uniformbackground such as a green screen, and generating a raw object image ata respective rotational angle of the turntable against the uniformbackground; performing data processing on the raw object images to matchimage characteristics of the raw object images to image characteristicsof the first camera to create final object images as a basis for theaugmented set of images.
 9. The method for generating an augmented setof images according to claim 8, further comprising: creating theaugmented set of images by adding the final object images to fieldimages collected with the first camera.
 10. The method for generating anaugmented set of images according to claim 8, wherein a set of finalobject images are the augmented set of images.
 11. The method forgenerating an augmented set of images according to claim 8, wherein theoptical and electronic characteristics of a first camera are chosen fromthe group comprising pixel pitch, pixel count, f/#, and FOV.
 12. Themethod for generating an augmented set of images according to claim 8,wherein the step of performing data processing includes modifying theraw object images to match a pixel intensity balance of the firstcamera.
 13. The method for generating an augmented set of imagesaccording to claim 8, wherein the step of performing data processingincludes removing a background in the raw object images.
 14. The methodfor generating an augmented set of images according to claim 13, whereinthe step of performing data processing further includes adding areplacement background in the raw object images after the background isremoved.
 15. The method for generating an augmented set of imagesaccording to claim 8, wherein the step of performing data processingincludes modifying the raw object images to simulate different ranges.